Papers with hyper-parameter tuning
Domain Adaptation with BERT-based Domain Classification and Data Selection (D19-61)
Copied to clipboard
| Challenge: | Modern deep neural models with millions of parameters can easily adapt to a new learning task and dataset when enough supervision is given. |
| Approach: | They propose a domain adaptation framework based on curriculum learning and domain-discriminative data selection. |
| Outcome: | The proposed framework outperforms discrepancy-based methods on transfer tasks while consuming only fraction of training budget. |
Azimuth: Systematic Error Analysis for Text Classification (2022.emnlp-demos)
Copied to clipboard
Gabrielle Gauthier-melancon, Orlando Marquez Ayala, Lindsay Brin, Chris Tyler, Frederic Branchaud-charron, Joseph Marinier, Karine Grande, Di Le
| Challenge: | Azimuth is an open-source tool to perform error analysis for text classification. |
| Approach: | They propose an open-source tool to perform error analysis for text classification . they propose to combine a range of ML techniques to facilitate the error analysis stage . |
| Outcome: | The proposed approach helps AI practitioners discover and address areas where the model does not generalize by leveraging and integrating a range of ML techniques. |
DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulations (2022.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation metrics for incoherent dialogues are insufficient to accurately reflect incohérence . despite the effectiveness of large pretrained language models, not everyone is into this type of work. |
| Approach: | They propose a Dialogue coherence Evaluation metric that uses Abstract Meaning Representation to apply semantic-level Manipulations for incoherent (negative) data generation. |
| Outcome: | The proposed evaluation metric achieves higher correlations with human judgments compared to baseline methods on dialog datasets by significant margins. |
A Fused Gromov-Wasserstein Framework for Unsupervised Knowledge Graph Entity Alignment (2023.findings-acl)
Copied to clipboard
| Challenge: | Entity alignment is the task of identifying corresponding entities across different knowledge graphs (KGs). |
| Approach: | They propose an unsupervised entity alignment framework that leverages the Fused Gromov-Wasserstein distance to compare KG semantics and KG structural information. |
| Outcome: | The proposed framework surpasses 21 competitive baselines, including cutting-edge methods, without supervision or hyper-parameter tuning. |
AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for Retrieval-Augmented Generation (2024.findings-emnlp)
Copied to clipboard
Jia Fu, Xiaoting Qin, Fangkai Yang, Lu Wang, Jue Zhang, Qingwei Lin, Yubo Chen, Dongmei Zhang, Saravan Rajmohan, Qi Zhang
| Challenge: | Recent advances in Large Language Models have transformed ML/AI development . a reevaluation of AutoML principles for Retrieval-Augmented Generation (RAG) systems is needed. |
| Approach: | They propose a framework for hyper-parameter tuning and a hierarchical MAB method for efficient exploration of large search spaces. |
| Outcome: | The proposed framework outperforms baseline methods in more challenging optimization scenarios. |
AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learning (N19-1)
Copied to clipboard
| Challenge: | Multi-task learning is an inductive transfer mechanism that leverages information from related tasks to improve the primary model's generalization performance. |
| Approach: | They propose a multitask learning pipeline that finds relevant auxiliary tasks and learns their mixing ratio. |
| Outcome: | The proposed model can find relevant auxiliary tasks and learn their mixing ratio . the proposed model achieves significant performance boosts on several primary tasks . |
Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Data (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Reinforcement Learning from Human Feedback (RLHF) relies on complex methodologies like Proximal Policy Optimization (PPO) that require extensive hyper-parameter tuning and pose challenges in sample efficiency and stability. |
| Approach: | They propose an innovative framework that leverages direct preference optimization techniques but extends them by estimating the conditionally optimal policy directly from the model’s responses. |
| Outcome: | The proposed framework matches and exceeds the effectiveness of Proximal Policy Optimization (PPO) in terms of convergence speed and alignment of model responses with human preferences. |
Towards Integration of Statistical Hypothesis Tests into Deep Neural Networks (P19-1)
Copied to clipboard
| Challenge: | Existing approaches for text classification are lexicallevel features with Naive Bayes or Support Vector Machines (SVM) . |
| Approach: | They propose a deep-learning model that uses label descriptions to train texts and their labels for multi-label and multi-class classification tasks. |
| Outcome: | The proposed model improves on one set with a high margin and on all other sets with competitive results. |
Robustness of Named-Entity Replacements for In-Context Learning (2023.findings-emnlp)
Copied to clipboard
Saeed Goodarzi, Nikhil Kagita, Dennis Minn, Shufan Wang, Roberto Dessi, Shubham Toshniwal, Adina Williams, Jack Lanchantin, Koustuv Sinha
| Challenge: | Modern large language models perform in-context learning, where query- answer demonstrations are shown before the final query. |
| Approach: | They propose to use in-context learning to prompt queries before they are answered . they find that the choice of demonstrations can affect model performance . |
| Outcome: | The proposed model performance improves on named entity replacements across three reasoning tasks and two popular LLMs. |
Fine-Tuning Language Models on Multiple Datasets for Citation Intention Classification (2024.findings-emnlp)
Copied to clipboard
Zeren Shui, Petros Karypis, Daniel Karls, Mingjian Wen, Saurav Manchanda, Ellad Tadmor, George Karypis
| Challenge: | Prior research has shown that pretrained language models (PLMs) can achieve state-of-the-art performance on CIC benchmarks. |
| Approach: | They propose a multi-task learning framework that fine-tunes pretrained language models on a dataset of primary interest together with multiple auxiliary CIC datasets to take advantage of additional supervision signals. |
| Outcome: | The proposed framework outperforms current state-of-the-art models on small datasets while aligning with the best-performing model on a large dataset. |
Towards Aligning Language Models with Textual Feedback (2024.emnlp-main)
Copied to clipboard
| Challenge: | Using textual feedback, language models can be trained to learn from textual inputs. |
| Approach: | They propose an approach that aligns language models with user preferences expressed in text. |
| Outcome: | The proposed approach outperforms PPO on toxicity reduction, summarization, and dialog response tasks while achieving the same performance with only 20% of the samples. |